Use time-of-flight workflow in Dream data reduction #125

nvaytet · 2025-01-27T14:39:16Z

In this PR, we incorporate the time-of-flight computation workflow into the DREAM data reduction workflow.

This results in much improved results where peaks in d-spacing are narrow and symmetrical, and lines are vertical along the 2theta dimension.

Before:

After:

…-spacing

jl-wynen · 2025-01-30T15:24:16Z

docs/user-guide/dream/dream-data-reduction.ipynb

   "metadata": {},
   "outputs": [],
   "source": [
-    "grouped_dspacing = workflow.compute(IofDspacingTwoTheta)\n",
+    "grouped_dspacing = workflow.compute(IofDspacingTwoTheta).bins.concat(\"event_time_zero\")\n",


Does this mean that the result has a new dimension?

Yes, we expect NeXus data to have the event_time_zero dimension, because it records events from more than one pulse.
The data we have in the geant4/mcstas files isn't very realistic, it has values for tof that exceed 71ms.

But shouldn't the time have been removed at some point during conversion to d-spacing?

This is gone after the latest update. We no longer bin the data in event_time_zero, we keep it as an event coord.

jl-wynen · 2025-01-30T15:25:34Z

src/ess/dream/io/geant4.py

+    # Add a event_time_zero coord for each bin, but not as bin edges,
+    # as all events in the same pulse have the same event_time_zero, hence the `[:2]`
+    da.coords["event_time_zero"] = (
+        sc.scalar(1730450434078980000, unit="ns").to(unit="us") + da.coords["tof"]


What is this number?

It's a random date in the past. The actual value does not matter.
I added a comment.

jl-wynen · 2025-01-30T15:26:06Z

src/ess/dream/io/geant4.py

+
+    period = (1.0 / sc.scalar(14.0, unit="Hz")).to(unit="us")
+    # Bin the data into bins with a 71ms period
+    da = da.bin(tof=sc.arange("tof", 3) * period)


Why 2 bins? Does it just so happen that the simulation contains 2 pulses?

It is 2 bins because the tofs spill over into the next pulse, but not in the one after that.
But in principle, you could have that in another file.
I changed to compute a npulses based on the max tof and the period.

jl-wynen · 2025-01-30T15:27:02Z

src/ess/powder/conversion.py

+def compute_detector_time_of_flight(
+    detector_data: DetectorData[RunType], tof_workflow: TofWorkflow
+) -> TofData[RunType]:
+    wf = tof_workflow.pipeline.copy()


How expensive is this? Does it copy the cached intermediate results?

I always assumed it made a cheap shallow copy?
I think it's using https://networkx.org/documentation/stable/reference/classes/generated/networkx.Graph.copy.html?

jl-wynen · 2025-01-30T15:28:29Z

src/ess/powder/conversion.py

+def set_monitor_zeros_to_nan(
+    monitor: TofMonitorData[RunType, MonitorType],
+) -> TofMonitorDataZerosToNan[RunType, MonitorType]:
+    inds = monitor.values == 0.0
+    monitor.values[inds] = np.nan
+    return TofMonitorDataZerosToNan[RunType, MonitorType](monitor)


Looking back at the discussion about granularity, can this be merged with the above provider? Do we need to expose the data without NaNs here? Or should it be part of convert_monitor_to_wavelength?

Also, this provider modifies its input which can lead to surprises down the line. Avoiding this would be easiest by merging it into compute_monitor_time_of_flight.

Yeah, I changed my mind at least twice on this one.
In the end, I made it into separate steps because I thought maybe not all workflows or instruments would want to NaN the zero counts, and it would make it easier to insert a different provider.

But I agree about making the graph more messy and modifying the data in place.

jl-wynen · 2025-01-30T15:33:44Z

src/ess/snspowder/powgen/workflow.py

+    detector_data:
+        Data from the detector.
+    """
+    return TofData[RunType](detector_data)


Can we avoid this dummy provider by changing extract_raw_data to return TofData instead of DetectorData?

jl-wynen · 2025-02-06T13:37:56Z

src/ess/dream/io/geant4.py

+    npulses = int((da.bins.coords["tof"].max() / period).value)
+    da = da.bin(tof=sc.arange("tof", npulses + 1) * period)


Suggested change

npulses = int((da.bins.coords["tof"].max() / period).value)

da = da.bin(tof=sc.arange("tof", npulses + 1) * period)

npulses = int((da.bins.coords["tof"].max() / period).ceil().value)

da = da.bin(tof=sc.arange("tof", npulses) * period)

So that npulses actually is the number of pulses. Without this, I think [:npulses] below is incorrect.

…ble manually

nvaytet · 2025-02-14T13:39:13Z

@jl-wynen see update. It's now cleaner after the latest changes in essreduce.

The CI will fail until we release essreduce and update the deps.
(needs scipp/essreduce#180 and scipp/essreduce#189)

nvaytet · 2025-02-17T15:14:38Z

I also updated the use of metadata utilities after they were updated in scippneutron.

EDIT: Hmm it seems I should have waited for #124 ...

jl-wynen · 2025-02-20T09:23:30Z

docs/user-guide/dream/dream-make-tof-lookup-table.ipynb

Is this notebook supposed to be shown in the docs? If so it needs to be added to the ToC and needs some descriptions. If not, please move it somewhere else, e.g., /tools because it gets executed when building the docs.

Good point, I think I will actually move this to the essreduce docs, that shows in a generic way how to create a lookup table. I think moving it to a tools folder is good for now.

jl-wynen · 2025-02-20T09:26:42Z

docs/user-guide/dream/dream-make-tof-lookup-table.ipynb

+    "wf[time_of_flight.LookupTableRelativeErrorThreshold] = 0.02\n",
+    "# wf[time_of_flight.PulsePeriod] = 1.0 / sc.scalar(14.0, unit=\"Hz\")\n",
+    "# wf[time_of_flight.PulseStride] = 1\n",
+    "# wf[time_of_flight.PulseStrideOffset] = None"


Suggested change

"wf[time_of_flight.LookupTableRelativeErrorThreshold] = 0.02\n",

"# wf[time_of_flight.PulsePeriod] = 1.0 / sc.scalar(14.0, unit=\"Hz\")\n",

"# wf[time_of_flight.PulseStride] = 1\n",

"# wf[time_of_flight.PulseStrideOffset] = None"

"wf[time_of_flight.LookupTableRelativeErrorThreshold] = 0.02"

I uncommented them instead.

jl-wynen · 2025-02-20T09:30:16Z

src/ess/dream/data.py

+
+
+def tof_lookup_table_high_flux() -> str:
+    """Path to a HDF5 file containing a lookup table for high-flux ToF."""


How was this created? using McStas or ToF? (Please explain in the docstring)

src/ess/dream/io/geant4.py

src/ess/powder/types.py

…ble was created

Co-authored-by: Jan-Lukas Wynen <[email protected]>

…of-workflow

nvaytet added 11 commits January 15, 2025 15:58

incorporate tof computation in dream reduction workflow

fa91262

start making a wrapper provider

6ec89f9

use the tof workflow in the dream reduction

ad70023

start fixing tests

4f2ef80

mask zero counts in monitors to avoid infinite values everywhere in d…

0b57f54

…-spacing

use nanmin as bin edges can be nan after tof lookup

85f984d

update more tests

6b34a2c

fix last tests

5d4c819

update essreduce version and add tof to deps

3150b42

add tof to nightly deps

2c5aa16

update deps

9a83342

nvaytet marked this pull request as ready for review January 30, 2025 12:15

fix language in notebook

3bdb149

nvaytet requested a review from jl-wynen January 30, 2025 12:33

nvaytet marked this pull request as draft January 30, 2025 12:35

nvaytet added 3 commits January 30, 2025 14:08

add dummy tof provider for powgen to skip the tof workflow step

3f83797

fix powgen tests

e214f05

fix dream notebook

3c7e826

nvaytet marked this pull request as ready for review January 30, 2025 14:55

add tof where it was missing

5927e57

jl-wynen reviewed Jan 30, 2025

View reviewed changes

nvaytet added 5 commits January 31, 2025 10:47

make tof a runtime dependency

7b145e1

explain better in assemble data and make it robust to more pulses

0164cf9

merge NaN step in workflow and remove dummy provider in powgen

5ff4ec8

remove left over providers from __init__

f245f0e

fix workflow by removing DetectorData

e87d443

jl-wynen reviewed Feb 6, 2025

View reviewed changes

start using new version of tof workflow after update

d7c8ec0

nvaytet marked this pull request as draft February 13, 2025 17:10

use correct choppers and make Ltotal separate for detectors and monitors

e8a3267

nvaytet added 3 commits February 14, 2025 13:38

static analysis

95147fb

switch to using a filename parameter instead of having to load the ta…

ebcefaa

…ble manually

fix dream geant4 tests

a274a34

nvaytet added 4 commits February 17, 2025 15:11

bump essreduce version

b55f608

merge main branch and update deps to get latest essreduce

bbabb72

fix init file

6ebbd44

fix workflow widgets notebook

7a814df

nvaytet marked this pull request as ready for review February 17, 2025 14:47

nvaytet added 2 commits February 17, 2025 16:06

fix metadata stuff from new scippneutron

579d87b

fix orcid_id in tests

1e36615

nvaytet added 3 commits February 17, 2025 16:31

update type in POWGEN notebook

55ec664

Merge branch 'main' into tof-workflow

a9bc401

fix dependency files

017d31c

nvaytet mentioned this pull request Feb 18, 2025

Add stap demo dream notebook #117

Draft

Merge branch 'main' into tof-workflow

ead136d

jl-wynen reviewed Feb 20, 2025

View reviewed changes

nvaytet and others added 4 commits February 20, 2025 13:06

move notebook to create tof lookup table and add comment about how ta…

f2e9894

…ble was created

Apply suggestions from code review

b134104

Co-authored-by: Jan-Lukas Wynen <[email protected]>

Merge branch 'tof-workflow' of github.com:scipp/essdiffraction into t…

85597b8

…of-workflow

Apply automatic formatting

5c4b3f8

jl-wynen approved these changes Feb 20, 2025

View reviewed changes

nvaytet merged commit bcd4e48 into main Feb 20, 2025
4 checks passed

nvaytet deleted the tof-workflow branch February 20, 2025 14:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use time-of-flight workflow in Dream data reduction #125

Use time-of-flight workflow in Dream data reduction #125

nvaytet commented Jan 27, 2025 •

edited

Loading

jl-wynen Jan 30, 2025

nvaytet Jan 31, 2025

jl-wynen Jan 31, 2025

nvaytet Feb 14, 2025

jl-wynen Jan 30, 2025

nvaytet Jan 31, 2025

jl-wynen Jan 30, 2025

nvaytet Jan 31, 2025

jl-wynen Jan 30, 2025

nvaytet Jan 31, 2025

jl-wynen Jan 30, 2025

nvaytet Jan 31, 2025

jl-wynen Jan 30, 2025

jl-wynen Feb 6, 2025

nvaytet commented Feb 14, 2025 •

edited

Loading

nvaytet commented Feb 17, 2025 •

edited

Loading

jl-wynen Feb 20, 2025

nvaytet Feb 20, 2025

jl-wynen Feb 20, 2025

nvaytet Feb 20, 2025

jl-wynen Feb 20, 2025

nvaytet Feb 20, 2025

		npulses = int((da.bins.coords["tof"].max() / period).value)
		da = da.bin(tof=sc.arange("tof", npulses + 1) * period)



		def tof_lookup_table_high_flux() -> str:
		"""Path to a HDF5 file containing a lookup table for high-flux ToF."""

Use time-of-flight workflow in Dream data reduction #125

Use time-of-flight workflow in Dream data reduction #125

Conversation

nvaytet commented Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvaytet commented Feb 14, 2025 • edited Loading

nvaytet commented Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvaytet commented Jan 27, 2025 •

edited

Loading

nvaytet commented Feb 14, 2025 •

edited

Loading

nvaytet commented Feb 17, 2025 •

edited

Loading